How to verify ground truth tasks

Seung Lab

This document describes the procedure a reviewer should follow when inspecting each type of annotation task.

Cell segmentation

VAST inspection

  1. Check borders
  1. Spend a few minutes cycling up and down through the stack to get a sense of how accurate the coloring is close to the borders. If there are a significant amount of bleeding (coloring over the border) or undercoloring (not coloring all the way to the border), make a note of the affected segments.
  1. Check there are no mergers & no splits
  1. Inspect each segment one-by-one: select a segment, then scroll through the relevant portion of the stack.
  2. If a segment is improperly joined with another segment, make a note of the affected segments.
  3. If a segment is not properly merged with another segment, make a note of the affected segments.
  1. Check there are no missing segments
  1. Cycle through the entire stack paying careful attention to the areas that don’t have any segments. If there is a segment that was missed, make a note of it.
  1. Decide whether to verify the task.
  1. If there are no border issues, no mergers or splits, and no missed segments, verify the task. Let the original annotator know they did a good job.
  2. If there are border issues, mergers or splits, or missed segments, do not verify the task. Let the original annotator know where they need to make corrections.

*Experts don’t necessarily need to correct their own mistakes, they just need to see them

Omni inspection

  1. Cycle through all the segments and inspect that for each:
  1. Check that it doesn’t contain a merge error
  2. Check that it’s not part of a split error
  3. Check that there’s no undercoloring
  4. Check that there’s no overcoloring
  5. Check that there’s no hanging pixels that weren’t erased completely
  6. Check that well-identified neighboring segments aren’t totally skipped

Synapse labeling

Mitochondria labeling

Semantic segmentation

Vesicle cloud labeling

Myelin labeling

Appeal process

  1. Acquire assistance from a Verified Researcher